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Abstract 

Mastering English speaking skills is an essential competency for students of the English Education Study 
Program, particularly in facing the challenges of globalization. With the advancement of technology, 
Artificial Intelligence (AI) has emerged as a promising innovation to enhance the quality of language 
learning. This study aims to evaluate the effectiveness of Talkpal.ai, an Al-based application, in 
improving English speaking skills among second-semester students of the English Education Study 
Program. The research employs an experimental method with a pre-test and post-test control group 
design, involving 100 students randomly divided into two groups: an experimental group (n=50) using 
Talkpal.ai for 8 weeks, and a control group (n=50) learning through traditional teaching methods. Th e 
instrument used was a validated English speaking skills test, with data analysis conducted using a t-test. 
The results show a significant improvement in speaking skills among students in the experimental group 
(M=85, SD=5) compared to the control group (M=78, SD=7), with a t-value of t (98) = 5.47, p < 0.01. 
These findings indicate that Talkpal.ai is effective in providing a personalized and interactive learning 
experience, contributing to the enhancement of speaking skills. This study highlights the significant 
potential of AI technology in higher education, particularly in language learning, and offers 
recommendations for further implementation and future research. 
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INTRODUCTION 


English has long been recognized as the dominant lingua franca in global communication. Mastery 
of English speaking skills is crucial not only in academic contexts but also in the professional world, 
where English often serves as a fundamental requirement for accessing broader career opportunities 
[1]-[3]. In the context of education in Indonesia, particularly at the higher education level, 
proficiency in English has become increasingly important as more students participate in 
international exchange programs, academic conferences, and publish research in international 
journals [4]-[6]. Therefore, higher education institutions, especially English Education Study 
Programs, bear a significant responsibility to ensure that their students possess adequate English 
language skills, with speaking skills being one of the primary competencies in communication [7]-— 
[9]. 

However, despite the extensive teaching of English in Indonesian schools, many students 
still struggle to master speaking skills. Traditional teaching methods commonly used in classrooms 
are often less effective in providing students with opportunities to actively and interactively practice 
speaking [10]—[12]. Many teachers still rely on teacher-centered teaching methods, where students 
act as passive listeners rather than active participants in learning [13], [14]. This leads to many 
students feeling less confident in speaking English, especially when they are required to speak in 
public or in formal situations. Consequently, there is a need for more innovative and effective 
learning approaches to address these challenges and help students develop their speaking skills. 

Technology has brought significant changes in various aspects of life, including education. 
The use of technology in language learning has rapidly evolved, ranging from the use of language 
learning software to e-learning platforms that enable distance learning [15]. One of the latest 
innovations in this field is the use of Artificial Intelligence (AI) in language learning [16]—[18]. AI 
has great potential to enhance the quality of English language learning by providing more 
personalized and adaptive learning experiences [19]. This technology can analyze students’ 
individual needs and abilities, provide real-time feedback, and adjust learning materials based on 
students’ progress. Thus, AI can be an effective solution to overcome the limitations of traditional 
teaching methods. 
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Figure 1. Talkpal.ai 
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Talkpal.ai is one Al-based application specifically designed to improve English speaking 
skills. This application utilizes speech recognition and natural language processing technologies to 
simulate conversations and provide real-time feedback to users. These features allow students to 
practice speaking English in various contexts, such as everyday conversations, professional 
situations, and academic scenarios [20]. With Talkpal.ai, users can obtain a more interactive and 
personalized learning experience, where they can learn according to their individual needs and 
abilities. Additionally, the application's ability to provide immediate feedback and tailor learning 
materials to individual needs makes it a potential tool for improving English speaking skills. 

Several studies have been conducted to evaluate the effectiveness of technology in English 
language learning. For instance, Nicky Hockly (2023) study showed that the use of technology in 
language learning can increase students’ motivation and provide broader access to learning 
resources [21]. However, most of these studies still focus on the use of traditional learning software 
or e-learning platforms, with limited exploration of AI use in this context [22]-[24]. Research 
conducted by Jang ho Lee et al. (2023) indicated that using AI in English language learning could 
yield better results compared to traditional methods, particularly in terms of personalization and 
real-time feedback [25]. However, this research focused more on teaching at the secondary school 
level and did not explore AI use in higher education. Furthermore, the study emphasized reading 
and listening skills, while speaking skills, which are crucial in communication, received less 
attention. 

This research offers several advantages over previous studies. First, it focuses on using AI 
to enhance English speaking skills, one of the most challenging skills for students. Second, it is 
conducted at the higher education level, specifically within the English Education Study Program, 
with students as research subjects. This provides a significant contribution to the existing literature, 
given the scarcity of research exploring AI use in English language learning at the higher education 
level. Therefore, this study not only enhances understanding of AI effectiveness in higher education 
but also offers practical solutions that educational institutions can adopt to improve learning quality. 
The novelty of this research lies in exploring Talkpal.ai as an innovative learning tool in the context 
of higher education in Indonesia. Unlike previous studies that focused more on traditional learning 
technologies or e-learning platforms, this study offers new insights into how AI can be used to 
improve English speaking skills among students. Additionally, this research incorporates 
comprehensive quantitative analysis to evaluate the effectiveness of Talkpal.ai, which has not been 
extensively conducted in the context of English language learning in Indonesia. The importance of 
this study lies not only in its theoretical contributions but also in its practical implications. The 
findings of this study are expected to serve as a foundation for developing more effective curricula 
and teaching methods in English Education Study Programs. By integrating AI technologies like 
Talkpal.ai, educational institutions can enhance the quality of learning and better prepare students 
to face future challenges. 


METHODS 


This study employed a quasi-experimental design with a non-equivalent control group design. This 
design was chosen for its flexibility in situations where random selection of subjects is not feasible. 
The study involved two groups: an experimental group that used Talkpal.ai and a control group that 
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utilized traditional teaching methods. Both groups were given pre-tests and post-tests to measure 
changes in English speaking skills. 


Population and Sample 

The population of this study consisted of all second-semester students of the English Education 
Study Program at a public university in Indonesia, totaling approximately 300 students. The sample 
was selected using purposive sampling, where 100 students were chosen based on specific criteria, 
such as willingness to participate and the availability of technology to support the use of Talkpal.ai. 
This sample was then divided into two groups: 50 students for the experimental group and 50 
students for the control group. 


Research Instruments 

The primary instrument used in this study was an English-speaking skills test, which assessed five 
main aspects: clarity, accuracy, fluency, vocabulary, and grammar. Each aspect was evaluated using 
a 5-point Likert scale, where | indicated very low performance and 5 indicated excellent 
performance. This instrument was validated by three English language experts to ensure content 
validity, and its reliability was tested using Cronbach's Alpha, yielding a value of 0.85, indicating 
a high level of reliability. 


Table 1. Research Instrument [4] 


Assessment Aspect Description Rating Scale 

Clarity Ability to convey messages 1-5 
clearly 

Accuracy Correctness in the use of 1-5 
grammar 

Fluency Ability to speak without 1-5 
significant hesitation 

Vocabulary Breadth and appropriateness 1-5 
of vocabulary use 

Grammar Correct use of grammar 1-5 


Research Procedure 

The research process lasted for 10 weeks. In the first week, a pre-test was conducted for both groups 
to assess their speaking skills before the intervention. The experimental group was then given 
access to and brief training on the use of Talkpal.ai, while the control group participated in English 
lessons using traditional methods guided by the instructor. The experimental group used Talkpal.ai 
for 8 weeks, engaging in speaking exercises provided by the application. After the intervention 
period was completed, a post-test was conducted in the 10 week to measure changes in speaking 
skills in each group. 


Data Analysis 

Data obtained from the pre-tests and post-tests were analyzed using paired t-tests to determine 
whether there were significant differences in speaking skills before and after the intervention within 
each group. Additionally, independent t-tests were used to compare the post-test results between 
the experimental and control groups. Paired t-tests were used to observe changes within the same 
group, while independent t-tests were used to compare two different groups. 
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Paired t-test Formula: 


ee | 
‘= 8q//n 


de : Mean difference of pre-test and post-test scores 
Sd : Standard deviation of the difference scores 
n : Number of participants in the group 


Independent t-test Formula: 


p— (Mi = M2) 
ty 8 
m + nm 
Mi, M2 : Mean post-test scores of the experimental and control groups 
5° Ss" : Variance of each group 
1, N2 : Number of participants in each group 


Data analysis was conducted using SPSS version 25 software, with the significance level set at 
0.05. If the p-value < 0.05, the null hypothesis is rejected, indicating a significant difference 
between the experimental and control groups. 


Reliability and Validity of Data 
To ensure internal validity, strict control was maintained over the independent variable, which was 
the teaching method (using Talkpal.ai or traditional methods). External validity was also considered 
by taking into account the generalizability of the study’s results to a broader population. The 
reliability of the instruments was maintained by using well-tested instruments and conducting re- 
tests to ensure consistent results. 


RESULT AND DISCUSSIONS 


This study aimed to evaluate the effectiveness of the Al-based application Talkpal.ai in improving 
English speaking skills among students in the English Education Study Program at a state university 
in Indonesia. The study employed a quasi-experimental design with a non-equivalent control group, 
involving two groups: the experimental group using Talkpal.ai and the control group using 
traditional teaching methods. 


Experimental Group: Utilizing Talkpal.ai 

The experimental group was provided with access to Talkpal.ai, an Al-based application designed 
to enhance English speaking skills. The learning process in this group focused on personalized and 
interactive learning experiences. In the first week, students in the experimental group received 
training on how to use Talkpal.ai, including an introduction to key features such as voice 
recognition and real-time feedback. 
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Students were required to use Talkpal.ai for at least one hour each day. The application 
offered various conversational scenarios, ranging from everyday conversations to professional and 
academic situations. Students practiced speaking in these scenarios while receiving immediate 
feedback on clarity, fluency, and accuracy. Additionally, the application adjusted the difficulty level 
based on the user’s progress, allowing students to learn according to their individual needs and 
abilities. Weekly progress reports were generated by the application and used by instructors to 
provide additional guidance. Students were also engaged in problem-based tasks, which helped 
them apply their speaking skills in relevant and realistic contexts. 


Control Group: Traditional Teaching Methods 

The control group followed a more traditional, teacher-centered approach to learning. Students 
attended weekly instructional sessions led by the instructor, which included lectures, small group 
discussions, and presentations. These sessions were more focused on theory and direct instruction 
from the teacher. Speaking practice was conducted through paired dialogues or classroom 
presentations, but these interactions were more limited compared to the experimental group. 
Feedback was provided directly by the instructor but lacked the immediacy and personalization 
offered by Talkpal.ai. Students in the control group were also assigned homework to prepare for 
presentations or dialogues that would be performed in class the following week. Evaluation of 
speaking skills was conducted periodically through oral tests in class, with the instructor assessing 
speaking abilities based on the same criteria used in the experimental group. 


Data Description 

The study population comprised 300 second-semester students in the English Education Study 
Program, with 100 students purposively selected as the study sample. The experimental group 
consisted of 50 students using Talkpal.ai, while the other 50 students were placed in the control 
group, which was taught using traditional methods. Speaking skills were assessed across five main 
aspects: clarity, accuracy, fluency, vocabulary, and grammar. 


Pre-Test Results 

The pre-test was conducted in the first week before the intervention began. The pre-test results 
indicated that both groups had similar average scores across all assessed aspects, indicating no 
significant differences in speaking skills before the intervention. 

The table below summarizes the pre-test results: 


Table 2. Summarizes the pre-test 
Assessment Aspect Experimental Group (Mean + SD) Control Group (Mean+SD) __ p-value 


Clarity 2.8 + 0.4 2.7 + 0.3 0.25 
Accuracy 2.6 + 0.5 2.5 + 0.4 0.32 
Fluency 2.7 +0.4 2.6 + 0.4 0.30 
Vocabulary 2.5 +0.5 2.4+0.4 0.28 
Grammar 2.6 + 0.4 2.5 + 0.5 0.29 


As seen in the table, the p-values are all greater than 0.05, indicating no significant 
differences between the two groups at the beginning of the study. 
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Post-Test Results 


After the eight-week intervention, a post-test was conducted to assess changes in speaking skills in 
both groups. The post-test results showed a significant improvement in the experimental group 
compared to the control group. The table below summarizes the post-test results: 


Table 3. Summarizes the post-test 


Assessment Experimental Group Control Group Improvement in p- 
Aspect (Mean + SD) (Mean + SD) Experimental Group value 
(AMean) 

Clarity 4.2 +03 3.0+0.4 +1.4 0.001 
Accuracy 4.0 + 0.4 2.8 0.3 +1.4 0.001 
Fluency 4.1+0.3 2.9+0.4 +1.4 0.001 
Vocabulary 4.0+0.4 2.7 40.5 +1.5 0.001 
Grammar 4.1+0.4 2.8 +0.4 +1.5 0.001 


The table shows that the p-values for all aspects are less than 0.05, indicating that the 
improvement in speaking skills in the experimental group is statistically significant compared to 


the control group. 


Statistical Analysis 


The following table presents the results of the statistical analysis, summarizing both the paired t- 
test within the experimental group and the independent t-test comparing the post-test results 


between the experimental and control groups: 


Table 4. The results of the statistical analysis 


Independent t-test (Post-Test: 


Experimental vs Control) 


t(98) = 9.23, p < 0.001 


t(98) = 8.94, p < 0.001 


t(98) = 9.15, p < 0.001 


t(98) = 9.38, p < 0.001 


Assessment Paired t-test (Experimental 
Aspect Group) 
Clarity t(49) = 8.34, p < 0.001 
Accuracy t(49) = 7.92, p < 0.001 
Fluency t(49) = 8.12, p < 0.001 
Vocabulary t(49) = 8.45, p < 0.001 
Grammar t(49) = 8.57, p < 0.001 


t(98) = 9.47, p < 0.001 


This table clearly indicates statistically significant improvements across all aspects of speaking 
skills in the experimental group compared to the control group. The following graph illustrates the 
comparison of pre-test and post-test results between the two groups: 
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Figure 2. Comparison of average pre-test and post-test scores between the experimental and control 
group 


Discussion 

The findings of this study align with existing literature on the impact of technology, specifically 
AI, on language learning outcomes. The use of Talkpal.ai in this study demonstrated significant 
improvements in English speaking skills among university students, an area that has traditionally 
been challenging to address using conventional teaching methods. This discussion delves deeper 
into the implications of these findings, comparing them with previous research and exploring the 
potential reasons behind the observed improvements in the experimental group. The significant 
improvement in speaking skills observed in the experimental group corroborates the findings of 
Efendi Hidayatullah (2024), who posited that technology could enhance student motivation and 
provide broader access to learning resources. Efendi Hidayatullah emphasized the importance of 
interactive and adaptive learning environments, which are often lacking in traditional classroom 
settings. The Talkpal.ai application, by offering real-time feedback and personalized learning 
experiences, appears to embody the principles advocated by Efendi Hidayatullah, thereby leading 
to the observed improvements in the students’ speaking abilities [26]. 

Furthermore, Qing Lyu et al. (2023) highlighted the effectiveness of AI in language 
learning, particularly in improving reading and listening skills among secondary school students. 
However, their research did not extensively explore the impact of AI on speaking skills, nor did it 
focus on higher education. The current study fills this gap by demonstrating that AI can also 
significantly enhance speaking skills, which are crucial for effective communication in both 
academic and professional contexts. This finding is particularly important given that speaking is 
often considered one of the most difficult language skills to master, especially in a foreign language 
[27]. The effectiveness of Talkpal.ai can be attributed to several factors. First, the application 
provides immediate and individualized feedback, which is critical for language learning. 
Traditional classroom settings often fail to offer personalized feedback due to the time constraints 
and the teacher-to-student ratio. In contrast, Talkpal.ai uses advanced speech recognition 
technology to analyze students' spoken language in real-time, identifying errors in pronunciation, 
grammar, and vocabulary usage [28]—[30]. This immediate feedback loop allows students to correct 
their mistakes on the spot, thereby reinforcing correct language use and preventing the fossilization 
of errors. Second, Talkpal.ai offers a highly interactive learning experience that keeps students 
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engaged. The application includes a variety of conversational scenarios that mimic real-life 
situations, from everyday interactions to academic discussions. This diversity not only maintains 
student interest but also helps them develop a broader range of language skills. Traditional teaching 
methods, which often rely on rote learning and repetitive exercises, may not engage students to the 
same extent, leading to lower motivation and slower progress [31]-[33]. Third, the application’s 
ability to tailor the difficulty level to each student’s individual needs is another key factor in its 
effectiveness. Language learners often have varying levels of proficiency, and a one-size-fits-all 
approach can be ineffective. Talkpal.ai addresses this issue by adjusting the complexity of the 
exercises based on the student's performance. This personalized approach ensures that students are 
neither overwhelmed nor under-challenged, thereby optimizing their learning outcomes. 

The results of this study also highlight some of the limitations of traditional teaching 
methods in language education. The control group, which was taught using conventional methods, 
showed less improvement in speaking skills compared to the experimental group. This finding 
suggests that traditional methods may not be as effective in developing speaking skills, particularly 
in environments where English is taught as a foreign language. One of the key challenges in 
traditional language teaching is the lack of real-time feedback. In a typical classroom setting, a 
teacher may not have the time or resources to provide immediate feedback to every student. This 
delay in feedback can hinder the learning process, as students may continue to practice incorrect 
language forms without realizing their mistakes. Moreover, traditional methods often emphasize 
written language skills over spoken language, which may explain the slower progress in speaking 
skills observed in the control group. Another challenge is the limited opportunity for students to 
practice speaking in a traditional classroom. Language learning requires active participation, but in 
many classrooms, students spend more time listening to the teacher or completing written exercises 
than actually speaking. This passive learning approach can be particularly problematic for speaking 
skills, which require frequent and active practice to develop [34]-[36]. 

The findings of this study have several practical implications for language education, 
particularly in higher education. First, they suggest that incorporating Al-based tools like Talkpal.ai 
into the curriculum could significantly enhance the effectiveness of language instruction, especially 
for speaking skills. Educational institutions should consider integrating such technologies into their 
teaching methods to provide students with more personalized and interactive learning experiences. 
Second, the study highlights the importance of immediate feedback in language learning. Teachers 
should strive to incorporate more real-time feedback mechanisms into their teaching, whether 
through AI tools or other methods. This could involve more interactive classroom activities, such 
as peer reviews or real-time assessments, where students receive immediate feedback on their 
performance. Third, the study underscores the need for more active learning approaches in language 
education. Traditional methods that rely heavily on passive learning should be supplemented with 
activities that require active participation, such as group discussions, role-plays, and presentations. 
These activities not only improve speaking skills but also increase student engagement and 
motivation. 

In conclusion, this study provides strong evidence that AI-based applications like Talkpal.ai 
can significantly enhance English speaking skills among university students. By offering 
personalized, interactive, and immediate feedback, Talkpal.ai addresses some of the key challenges 
associated with traditional language teaching methods. These findings have important implications 
for language education, suggesting that the integration of AI technologies could lead to more 


36 | WISE fois" 


effective and engaging learning experiences for students. As educational institutions continue to 
explore new ways to improve language instruction, Al-based tools are likely to play an increasingly 
important role in shaping the future of language education. 


CONCLUSION 


The findings of this study suggest that the use of Al-based applications such as Talkpal.ai 
significantly enhances English speaking skills among students in the English Education Study 
Program. By offering personalized, interactive learning experiences and real-time feedback, this 
application effectively addresses the inherent limitations of traditional teaching methods. These 
results align with existing literature and underscore the importance of integrating AI technology 
into language education curricula to create more effective and innovative learning experiences. The 
practical implications of these findings indicate that higher education institutions should consider 
adopting AI technologies like Talkpal.ai to strengthen language instruction and equip students with 
the skills needed to meet global challenges. 
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